Welcome to the final programming exercise of this course! For this week, your task is to adapt the Cats vs Dogs Class Activation Map ungraded lab (the second ungraded lab of this week) and make it generate saliency maps instead.
As discussed in the lectures, a saliency map shows the pixels which greatly impacts the classification of an image.
In this assignment you will be given prompts but less starter code to fill in in.
Let's begin by first downloading files we will be using for this lab.
# Download the same test files from the Cats vs Dogs ungraded lab
!wget -O cat1.jpg https://storage.googleapis.com/tensorflow-1-public/tensorflow-3-temp/MLColabImages/cat1.jpeg
!wget -O cat2.jpg https://storage.googleapis.com/tensorflow-1-public/tensorflow-3-temp/MLColabImages/cat2.jpeg
!wget -O catanddog.jpg https://storage.googleapis.com/tensorflow-1-public/tensorflow-3-temp/MLColabImages/catanddog.jpeg
!wget -O dog1.jpg https://storage.googleapis.com/tensorflow-1-public/tensorflow-3-temp/MLColabImages/dog1.jpeg
!wget -O dog2.jpg https://storage.googleapis.com/tensorflow-1-public/tensorflow-3-temp/MLColabImages/dog2.jpeg
# Download prepared weights
!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1kipXTxesGJKGY1B8uSPRvxROgOH90fih' -O 0_epochs.h5
!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1oiV6tjy5k7h9OHGTQaf0Ohn3FmF-uOs1' -O 15_epochs.h5
Please import:
# YOUR CODE HERE
import tensorflow as tf
import tensorflow_datasets as tfds
import keras
from keras.models import Sequential,Model
from keras.layers import Dense,Conv2D,Flatten,MaxPooling2D,GlobalAveragePooling2D
from keras.utils import plot_model
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import cv2
Required: Use Tensorflow Datasets to fetch the cats_vs_dogs dataset.
as_supervised flag to create (image, label) pairs.Optional: You can create validation and test sets from the remaining 20% of the train split of cats_vs_dogs (i.e. you already used 80% for the train set). This is if you intend to train the model beyond what is required for submission.
# Load the data and create the train set (optional: val and test sets)
# YOUR CODE HERE
train_data = tfds.load('cats_vs_dogs', split='train[:80%]', as_supervised=True)
validation_data = tfds.load('cats_vs_dogs', split='train[80%:90%]', as_supervised=True)
test_data = tfds.load('cats_vs_dogs', split='train[-10%:]', as_supervised=True)
Define a function that takes in an image and label. This will:
def augmentimages(image, label):
image = tf.cast(image, tf.float32)
image = (image/255)
image = tf.image.resize(image,(300,300))
return image, label
Use the map() and pass in the method that you just defined to preprocess the training set.
augmented_training_data = train_data.map(augmentimages)
This is already provided for you. Normally, you will want to shuffle the training set. But for predictability in the grading, we will simply create the batches.
# Shuffle the data if you're working on your own personal project
train_batches = augmented_training_data.shuffle(1024).batch(32)
train_batches = augmented_training_data.batch(32)
You'll define a model that is nearly the same as the one in the Cats vs. Dogs CAM lab.
Dense layer.do_salience() function from the lab and that works with one-hot encoded labels.units argument of the output Dense layer from 1 to 2, with one for each of the classes (i.e. cats and dogs).# YOUR CODE HERE
model = Sequential()
model.add(Conv2D(16,input_shape=(300,300,3),kernel_size=(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(32,kernel_size=(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64,kernel_size=(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(128,kernel_size=(3,3),activation='relu',padding='same'))
model.add(GlobalAveragePooling2D())
model.add(Dense(2,activation='softmax'))
model.summary()
Expected Output:
txt
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 300, 300, 16) 448
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 150, 150, 16) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 150, 150, 32) 4640
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 75, 75, 32) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 75, 75, 64) 18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 37, 37, 64) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 37, 37, 128) 73856
_________________________________________________________________
global_average_pooling2d (Gl (None, 128) 0
_________________________________________________________________
dense (Dense) (None, 2) 258
=================================================================
Total params: 97,698
Trainable params: 97,698
Non-trainable params: 0
_________________________________________________________________
Complete the do_salience() function below to save the normalized_tensor image.
def do_salience(image, model, label, prefix):
'''
Generates the saliency map of a given image.
Args:
image (file) -- picture that the model will classify
model (keras Model) -- your cats and dogs classifier
label (int) -- ground truth label of the image
prefix (string) -- prefix to add to the filename of the saliency map
'''
# Read the image and convert channel order from BGR to RGB
# YOUR CODE HERE
img = cv2.imread(image)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
# Resize the image to 300 x 300 and normalize pixel values to the range [0, 1]
# YOUR CODE HERE
img = cv2.resize(img, (300, 300)) / 255.0
# Add an additional dimension (for the batch), and save this in a new variable
# YOUR CODE HERE
img = np.expand_dims(img, axis=0)
# Declare the number of classes
# YOUR CODE HERE
num_classes=2
# Define the expected output array by one-hot encoding the label
# The length of the array is equal to the number of classes
# YOUR CODE HERE
expected_output=tf.one_hot([label]*img.shape[0], num_classes)
# Witin the GradientTape block:
# Cast the image as a tf.float32
# Use the tape to watch the float32 image
# Get the model's prediction by passing in the float32 image
# Compute an appropriate loss
# between the expected output and model predictions.
# you may want to print the predictions to see if the probabilities adds up to 1
# YOUR CODE HERE
with tf.GradientTape() as tape:
inputs = tf.cast(img, tf.float32)
tape.watch(inputs)
predictions = model(inputs)
loss = tf.keras.losses.categorical_crossentropy(
expected_output, predictions
)
print(predictions)
# get the gradients of the loss with respect to the model's input image
# YOUR CODE HERE
gradients = tape.gradient(loss, inputs)
# generate the grayscale tensor
# YOUR CODE HERE
grayscale_tensor = tf.reduce_sum(tf.abs(gradients), axis=-1)
# normalize the pixel values to be in the range [0, 255].
# the max value in the grayscale tensor will be pushed to 255.
# the min value will be pushed to 0.
# Use the formula: 255 * (x - min) / (max - min)
# Use tf.reduce_max, tf.reduce_min
# Cast the tensor as a tf.uint8
# YOUR CODE HERE
normalized_tensor = tf.cast(
255
* (grayscale_tensor - tf.reduce_min(grayscale_tensor))
/ (tf.reduce_max(grayscale_tensor) - tf.reduce_min(grayscale_tensor)),
tf.uint8,
)
# Remove dimensions that are size 1
# YOUR CODE HERE
normalized_tensor=tf.squeeze(normalized_tensor)
# plot the normalized tensor
# Set the figure size to 8 by 8
# do not display the axis
# use the 'gray' colormap
# This code is provided for you.
plt.figure(figsize=(8, 8))
plt.axis('off')
plt.imshow(normalized_tensor, cmap='gray')
plt.show()
# optional: superimpose the saliency map with the original image, then display it.
# we encourage you to do this to visualize your results better
# YOUR CODE HERE
# save the normalized tensor image to a file. this is already provided for you.
salient_image_name = prefix + image
normalized_tensor = tf.expand_dims(normalized_tensor, -1)
normalized_tensor = tf.io.encode_jpeg(normalized_tensor, quality=100, format='grayscale')
writer = tf.io.write_file(salient_image_name, normalized_tensor)
As a sanity check, you will load initialized (i.e. untrained) weights and use the function you just implemented.
If an error pops up when loading the weights or the function does not run, please check your implementation for bugs.
Please apply your do_salience() function on the following image files:
cat1.jpgcat2.jpgcatanddog.jpgdog1.jpgdog2.jpgCats will have the label 0 while dogs will have the label 1.
0.epoch0_salient.# load initial weights
model.load_weights('0_epochs.h5')
# generate the saliency maps for the 5 test images
# YOUR CODE HERE
do_salience('cat1.jpg',model,0,'epoch0_salient')
do_salience('cat2.jpg',model,0,'epoch0_salient')
do_salience('cat2.jpg',model,0,'epoch0_salient')
do_salience('cat2.jpg',model,1,'epoch0_salient')
do_salience('cat2.jpg',model,1,'epoch0_salient')
With untrained weights, you will see something like this in the output.
Use model.compile() to define the loss, metrics and optimizer.
Choose a loss function for the model to use when training.
model.compile() the ground truth labels from the training set are passed to the model as integers (i.e. 0 or 1) as opposed to one-hot encoded vectors.loss = 'loss_function_a').For metrics, you can measure accuracy.
For the optimizer, please use RMSProp.
0.001.# YOUR CODE HERE
model.compile(loss='SparseCategoricalCrossentropy',metrics=['accuracy'],optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.001))
Please pass in the training batches and train your model for just 3 epochs.
We have loaded pre-trained weights for 15 epochs so you can get a better output when you visualize the saliency maps.
# load pre-trained weights
model.load_weights('15_epochs.h5')
# train the model for just 3 epochs
# YOUR CODE HERE
model.fit(train_batches,epochs=3)
You will now use your do_salience() function again on the same test images. Please use the same parameters as before but this time, use the prefix salient.
# YOUR CODE HERE
do_salience('cat1.jpg', model, 0, "salient")
do_salience('cat2.jpg', model, 0, "salient")
do_salience('catanddog.jpg', model, 0, "salient")
do_salience('dog1.jpg', model, 1, "salient")
do_salience('dog2.jpg', model, 1, "salient")
You should see that the strong pixels are now very less than the ones you generated earlier. Moreover, most of them are now found on features within the pet.
Please run the cell below to zip the normalized tensor images you generated at 18 epochs. If you get an error, please check that you have files named:
Afterwards, please download the images.zip from the Files bar on the left.
from zipfile import ZipFile
!rm images.zip
filenames = ['cat1.jpg', 'cat2.jpg', 'catanddog.jpg', 'dog1.jpg', 'dog2.jpg']
# writing files to a zipfile
with ZipFile('images.zip','w') as zip:
for file in filenames:
zip.write('salient' + file)
print("images.zip generated!")
We have pre-trained weights generated at 95 epochs and you can see the difference between the maps you generated at 18 epochs.
!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=14vFpBJsL_TNQeugX8vUTv8dYZxn__fQY' -O 95_epochs.h5
model.load_weights('95_epochs.h5')
do_salience('cat1.jpg', model, 0, "epoch95_salient")
do_salience('cat2.jpg', model, 0, "epoch95_salient")
do_salience('catanddog.jpg', model, 0, "epoch95_salient")
do_salience('dog1.jpg', model, 1, "epoch95_salient")
do_salience('dog2.jpg', model, 1, "epoch95_salient")
Congratulations on completing this week's assignment! Please go back to the Coursera classroom and upload the zipped folder to be graded.